Senior Data Engineer with Databricks experience
Description
Role: Sr. Data Engineer - with Databricks experience
Location: Bangalore, India – Remote
Key Responsibilities
1. Databricks Development
Develop ETL/ELT pipelines using Spark (PySpark/Scala/SQL).
Work with Delta Lake for ACID-compliant data processing.
Build and optimize bronze–silver–gold architecture.
Handle large-scale batch and streaming data.
2. Data Integration
Ingest data from various sources (API, SQL Server, ADLS, Kafka, Event Hub, Service Bus).
Manage Databricks Jobs, Workflows, and Clusters.
Optimize pipelines for performance and cost.
3. Data Modeling & Storage
Create and optimize data models for analytics.
Work with Data Lake, Delta Lake, Hive Metastore.
Implement schema evolution, time travel, merge, and CDC.
4. Reporting & BI
Build dashboards in Power BI / Tableau / Looker.
Write optimized SQL for reporting layers.
Create semantic models, DAX measures, and calculated fields.
Publish reports and handle incremental refresh.
5. Governance & Quality
Implement CI/CD for Databricks using Git.
Apply data validation, logging, monitoring, and alerting.
Follow security best practices (RBAC, Key Vault, PAT tokens).
Excellent analytical and problem-solving abilities
Understanding of HR processes, compliance requirements, and industry best practices
Location: Bangalore, India – Remote
Key Responsibilities
1. Databricks Development
Develop ETL/ELT pipelines using Spark (PySpark/Scala/SQL).
Work with Delta Lake for ACID-compliant data processing.
Build and optimize bronze–silver–gold architecture.
Handle large-scale batch and streaming data.
2. Data Integration
Ingest data from various sources (API, SQL Server, ADLS, Kafka, Event Hub, Service Bus).
Manage Databricks Jobs, Workflows, and Clusters.
Optimize pipelines for performance and cost.
3. Data Modeling & Storage
Create and optimize data models for analytics.
Work with Data Lake, Delta Lake, Hive Metastore.
Implement schema evolution, time travel, merge, and CDC.
4. Reporting & BI
Build dashboards in Power BI / Tableau / Looker.
Write optimized SQL for reporting layers.
Create semantic models, DAX measures, and calculated fields.
Publish reports and handle incremental refresh.
5. Governance & Quality
Implement CI/CD for Databricks using Git.
Apply data validation, logging, monitoring, and alerting.
Follow security best practices (RBAC, Key Vault, PAT tokens).
Excellent analytical and problem-solving abilities
Understanding of HR processes, compliance requirements, and industry best practices